PGCDHEH7 msum 



BP 133 352 



SB 005 963 



TITLE 
POl DAIE 
NOTE 



ED13 PBXCS 
DESCEIPCOBS 



Kilbajie-Tlashr Marian I, 

The Titl€ 1 Evaluation Cycles 1965-1975, 

CApC 76] 

gp,; fap€r pEasaated at the Annual Seating of tha 
'ftmeti^an EaucatiDnal Eesearch A^aociation (SOth, 
Pranciaco^ CaliJornia, Apcll 19-23# 1976) 



San 



HC -41*67 Plus Po0tag€* 
3*Coapen0€tory Eaucatlon Pccgrans; Edueatioiial Change ; 
^Evaluation Hathodei ^Fedeial Brcgramsi ^Progran 
l^alaationt School Districts 

Cleveland Ptiblic Schoolei ^Eleaejitar^ Scconaars 
Iducataoii Act Title I; ISBA TitU I| Ohio 



ABSTRACT 

The a mergence of a segueac^ in eiralttation approaches 
at the local level flu ring the aecafle of Titli X is disc^ssea* iheae 
recurring pattariia telatefl to objecti^es^ €valuatioD aeaigiis and data 
coHaction mcaea retlect iwpact of both external factors (e^g«^ 
changes iii funding gaiaelines) and internal factocs {mtgmi changes in 
priority iieads) • However i periodic Ea-eieEgence o£ earlier approachas 
has been chaxact^ri^ad by increasiiigly iore coapreh ensile purposasr 
ifith icra specific identificatioa and inteixelatlonahip of process 
and proauct factors* The spiral saquenci thus generates "new'' 
conbinaticns end applicatioiie of "old" appioachea* (Aathot/SC) 
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mn TITLE I EVALUATION CYCLE' 
1965 1975 ^ 

The ten yeais of Title I havi v/itixesied ongoing changes in the 
natwe proceduxes of Tltl^ I evaluation at the local levil. Hiese 
changei r*eflect le spouses to needs and raquiremants both fzom witJiin the 
sehool my$t%m md extenial to it. Hie pattern of these ehmges emerges 
from a Mview of e /alijations of the past decade, lii the context of the 
circunistaiiGes and factors that.fDraied the Title I fabxie of which the 
evaluatioiis were one strand* This retrospective ajialysis of Title I 
evaluatioii in the Clavelmd Public Schooli yields three overlapping phases , 

PHASE Ij 1965 ' 1967 

The first phase covered the early years or roughly' 1965 to 1967* 
The avaluation procedures of this Indtial phase reflect the influeace of a 
variety of factors. The guideliaas for Tltli I were inteipreted as mandat- 
ing a general-aid approach, Consiqueiitly,^ for mmt of the Initial years' 
projects^ the only criterion for a pupil to participate was membership in 
a Title I schoDl* Services were also generalized In naturei enconipasiing 
such diffuse or indirect ^'treatTOeiits** as field trips , saargasbord-t)ps 
aftirschool prograniSi devalopmint of curriculum guidtSi etc* " Tht general-- 
ized quality of the services was accon^anled by objectives of similar gen- 
eral natura , 

ThB projects \imt& weathering the instEllstion pariod with all that 
that involves in securing and training staff, istabllshlng coimnunicatioii 
chmmels, clarifying roles, and obtaining or daveloping niatirials, Necassarily, 
concern had to center firit on ensurinj that procasi ©lejaants were' in place 
before product outcomes could beccme a reasonable e^ectatlon. 
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The task of rolt clarificatiori included the es tab iishnierit of 
credibility and trust as applied to ^valuators* Prpjact staff tanded to be 
defensive tdwa:rd or fearful of the consequences of evaluation, 

Evalyators, newly ''hirad mm defining objectives and devel- 

oping appropriate iristrUBieTits Md data. colleGtion choniiftls* Resources such 
as cDmputer programs were Ijeing sacuTed* Fonnats wei?a beinj evalv^d for 
:r#ports and foj feedback .procedures to project staff. 

Against this bacKdropj the evaluations of tUc Initial years wre 
charactexized by an enphasis on prucess assessraent dQcmentliig the CQniruenca 
between proposed ajid actijal pMCedural elements Cp^oviston of staffs facil- 
ities^ materials) and fuither doctffiieTitirig the perceptioas of project parti- 
cipants and staff. 

Data collectloa efflployad sit&^vlsltatiDn tesiaja interview Md 
qiitestionnatre data^ ajid iDcally^-constructed rating scales a^id attitude instTu- 
TCiiti. Stajidaxdized»teit data were vsed on only a liifllted lasls* Data 
malyjii was ps^inardly descriptive tii nature* 

For bath project in^lemintation mi evaluatloai tlie first ph-ase 
of Title I wai a period of gromd breaking. 

PBASB 2: 1968 - 1972 

Thm second phase j roughly spanning the period froin 1968 thrOTigh 
39 72| marked a sliift in evaluation focus and design* Hie eTOrgenca of nw 
diractioiis during the second phase repTessnted^ in parti a shift in eval- 
uation consistsiit ^ith the '*matiirati©ii'' af th« initial pxQjects--i,e,| pro- 
jects had moved heyoad the installation phwei thus ttiaMug yrgduct assess- 
Dient appropriate. The iecond-phase characteristics fuither refilectied th© 
'^maturation*' of evaluation roles and pTOcedures, Project staff perceptions 
of evaluation became more yositive««cr| at least, more toleiant and less 
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defensive. A closer worlcing relationship was established bstween evalmtors 
and project staff, with systemtic feidbaclc of evaluatioa results to pxoject 
st€ff, and with iiicreasiiig ricognltion that evaluation could contxibutf to 
project iwptQYBmmt as well as fulfill the requlrad legalities of fundiTig, 

En addition to these concomitants of nonnal giawth^ the sicDud 
phase Vai Influenced by a critleal external developMiit, The state -s Tein- 
tf^Tatation oS the guidelines validated a change fro^ a ganiTal-add approach 
to a categorical- aid approach. The change gava impetus to mwa definitive 
criteria of pupil eligibillt)^ fcr a giveti project* Necessarily^ this vfas 
accQupmied by imOTe specific definition of treatinent coinponaiits^ and more 
preclsi atateniants of both, process and product objectives* Several eaTlier 
prcj^ects of a general=aid qualit)^ wn& replaced by pT&jeets of the cat€- 
gp^tcal-aid tTpe. These latter projects had structures that permitted ram- 
doaiisad e5(paxljnental -control deaigms* 

Evaluations of this iecond phase reflected mi emphasis oa pupil- 
iCiiitered asiessmeiit of product outcomes* Classical aad laore complex 
rfatarch. designs wre applied, employing randomised a^^erimental- control 
groups , axasiinatioii of differential inpact of tiraatTnont variables ^ use of • 
Bmltivarista malysiSi etc, 

Dasigns required extensive use of st^dardlaad tests on a pre- 
post basis. There was continued use of _locally--coTiitriicted iastruniarits 
but v/ith detenmination of validity, reliability^ md factor structure, 

PHASE 3: 1973 - 

T^e thiffd phase--frQm 1973 to the prese4it--coiitiriued the f^mphasii 
on clearly specified process and product objectives, Tlie uie of standardized 
testlns on a pre-post basis remaiTiad m essential elemeiitj as did the colleC'- 
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tion of process and demographic data, and the docimentation of perceptions 
of staff, pupils MdpaTeTits. 

Two definabli Ganges in emphails emerged* First, randomized 
a^^STiineiital-cOTtrol designs, which had appeaTed in the previous phase, were 
no longer used. Instciad^ measures of student pTogxess wm tasad on stan- 
aids such as ^^^ected progress or pra-specified gain scorsSi SecQnd^ the 
assessinent of process dimensions raflected an increasing concern with the 
content of Instruational services being given to children* Observation of 
pTOjact activities received more intensive mttention, 

characteristics of the third-phase evaluations were associated 
with several factors. The demise of the classical expemnental-control 
desigri was a consequence of a directive from the state level requiring that 
pupils with the greatest need be given first priority for service. The 
attention fociised en in-depth process deflnltiDn was a natural consequence 
of those project avaluatlons which documented marked success in reaching 
product objectives. In effect^ systeinatic Qbservatlon data attenq)ted to 
laora clearly da fine the treatment which ms bringing abcat the successful 
out comes , 

Tha third phase was also marked by a close WDrking relationship 
betweeri evaluation and project staffs. In general ^ project staffs have come 
to vlav? evaluation as m activity in which they hav© Bn inportmt role, _ This 
role has been exercised through helping define objectives C^han proposals 
aie being written) ^ assisting in the refinement of instxuinents to effect 
greater congruence with project intentions^ considering evaluative feedback 
tc project staff as part of project activitieSi mnd incoi^orating data-col- 
lection procedures into ongoing project opiratlons* 
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In surmnary, the patterni emerging from the three evaluatioii phases 
reveal both chaage and maintenajice in evalmtion procedures* Data col lection 
modes of the earlieT yeaM--slte visitation^ observations, interviews --not 
only remain importaiit but have becqine niore systematic and ipeciflcally focused 
on key programnisLtte intentions* The concept of fomal pre -post aisessmant not 
only remains basic to Title I but has becoine an accepted alemcat of the, gen- 
eral instructional progrMimlng. 

Some of the initial problems remain as well* The basic evalnation 
question continuas to be a dileiroai hoi^r much would a child have progr©ssad 
without Title I? The guidelines preclude ming a randomized experimental-con- 
trol design. Even if such a design were permitted, other realitiai complicate 
the neat statistics. Thus, for example, high pupil mobility is a fact of 
life beyond the school's control. This mobility produces interruptigna in 
treatment and diffaiential treatment within a project* 

Our evaluation data document , the niarked in^rovements that result 
from delivery of saTvice on a systematic and long-term basis* Summaries of 
participants' pri-=pcst gains have been impxessivei but the gap betwem norm 
levals and parti ctpants* performance levels is difficult to close, Raducing 
or eliminating the gap requires acceleratioa sustained over mmy years, 

. A fourth phase of Title I avaluation may now be emerging* Onm 
contTibuting factor is the development of more sophisticated and systematic 
needs assessment procedures. Another is the mradatlng of a new imlt of measure- 
ment — the NCE or noxnial-curve equivalant--fOT docimentlng pupil change in • 
performmce. Still other factors are the lacraaslng pressura of project staff 
to use eriterloa-raferenced measures fox pioduct assessment^ mid the possibility 
of court -ordered busing. 
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